Modeling distributions of covariates, or density estimation, is a core challenge in unsupervised learning. However, the majority of work only considers the joint distribution, which has limited utility in practical situations. A more general and useful problem is arbitrary conditional density estimation, which aims to model any possible conditional distribution over a set of covariates, reflecting the more realistic setting of inference based on prior knowledge. We propose a novel method, Arbitrary Conditioning with Energy (ACE), that can simultaneously estimate the distribution p(xu | xo) for all possible subsets of unobserved features xu and observed features xo. ACE is designed to avoid unnecessary bias and complexity -- we specify densities with a highly expressive energy function and reduce the problem to only learning one-dimensional conditionals (from which more complex distributions can be recovered during inference). This results in an approach that is both simpler and higher-performing than prior methods. We show that ACE achieves state-of-the-art for arbitrary conditional likelihood estimation and data imputation on standard benchmarks.

Neural Information Processing Systems http://nips.cc/

Canonically, the variational principle suggests to prefer an expressive inference model so that the variational approximation is accurate.

Canonically, the variational principle suggests to prefer an expressive inference model so that the variational approximation is accurate.

Neural Information Processing Systems http://nips.cc/

We would like to thank the reviewers for their kind and thoughtful comments. Any attempt to mitigate particle degeneracy (e.g. Replicating Fig 1BD, we find similar, albeit less extreme results, with TMC always being faster than SMC. In particular, we have included Eq. 36 in the main text, and also included the corresponding choice of This should help to clarify that Eq. 11 applies to any directed graphical model (we have also included references In the example in Figure 1, we consider a model that does not have a chain-structure (see Appendix Figure 1A). IW AE performs arbitrarily badly due to the high-dimensionality of the state-space.

Importance sampling, which involves sampling from a probability density function (PDF) proportional to the product of an importance weight function and a base PDF, is a powerful technique with applications in variance reduction, biased or customized sampling, data augmentation, and beyond. Inspired by the growing availability of score-based generative models (SGMs), we propose an entirely training-free Importance sampling framework that relies solely on an SGM for the base PDF. Our key innovation is realizing the importance sampling process as a backward diffusion process, expressed in terms of the score function of the base PDF and the specified importance weight function--both readily available--eliminating the need for any additional training. We conduct a thorough analysis demonstrating the method's scalability and effectiveness across diverse datasets and tasks, including importance sampling for industrial and natural images with neural importance weight functions. The training-free aspect of our method is particularly compelling in real-world scenarios where a single base distribution underlies multiple biased sampling tasks, each requiring a different importance weight function. To the best of our knowledge our approach is the first importance sampling framework to achieve this.